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Abstract 

The  Moving  Target  Tutor  (MTT)  is  a  computer-based,  multimedia  tutoring  system  designed  to 
teach  people  an  important  skill  in  marksmanship:  shooting  moving  targets.  To  hit  moving 
targets,  a  person  must  be  able  to  quickly  retrieve  the  correct  Points  of  Aim  (PoA).  PoA  is  how  far 
to  lead  a  moving  target  based  on  the  range  of  the  target,  how  fast  it  is  moving,  and  the  angle 
between  shooter  and  the  target’s  heading.  MTT  uses  a  learning  theory  (Kim,  Koubek,  &  Ritter, 
2013)  implemented  in  an  architecture  called  D2P  (Ritter,  Yeh,  Cohen,  Weyhrauch,  Kim,  & 
Hobbs,  2013).  MTT  contains  approximately  two  to  three  hours  of  instructional  materials, 
including  instructional  text,  pictures,  audio,  videos,  and  quizzes.  These  materials  are  then 
presented  in  the  D2P  program  that  is  capable  of  handling  other  tutoring  content.  In  this  paper, 
we  present  the  effect  of  using  MTT  in  learning  PoA  by  university  students.  We  found  our 
participants  improved  in  accuracy  (more  than  tripled  their  ability  to  judge  PoA,  effect  size  = 
3.7)and  an  improvement  in  PoA  of)  and,  simultaneously,  reduced  the  time  to  determine  PoA. 
Although  the  participants  may  not  have  had  a  need  to  master  marksmanship  for  purposes  outside 
of  this  study,  this  report  still  provides  insights  about  the  effectiveness  of  using  MTT,  along  with 
D2P,  and  points  to  directions  for  future  improvements. 
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1.0  Introduction 

The  Moving  Target  Tutor  (MTT)  is  a  computer-based,  multimedia  tutoring  system  designed 
to  teach  people  an  important  skill  in  marksmanship:  shooting  moving  targets.  To  hit  moving 
targets,  a  person  must  be  able  to  retrieve  the  correct  Points  of  Aim  (PoA)  in  a  short  period  of 
time.  This  consists  in  learning  how  far  to  lead  a  moving  target  based  on  the  range  of  the 
target,  how  fast  it  is  moving,  and  the  angle  between  shooter  and  the  target’s  heading. 
Consequently,  determining  the  correct  PoA  is  the  result  of  combining  the  range,  speed,  and 
angle  of  a  moving  object. 

MTT  uses  a  learning  theory  (Kim,  Ritter,  &  Koubek,  2013)  that  emphasizes  how  declarative 
knowledge  is  transferred  into  procedural  knowledge,  and  how  these  types  of  knowledge  have 
different  learning  mechanisms  and  different  forgetting  mechanisms.  To  test  and  apply  this 
theory  we  created  the  Declarative  to  Procedural  (D2P)  tutoring  architecture  (Ritter  et  ah, 
2013). 

The  D2P/MTT  contains  roughly  two  to  three  hours  of  instructional  materials,  including 
instructional  text,  pictures,  audio,  videos,  and  quizzes.  These  materials  are  then  presented  in 
the  D2P  program  that  is  capable  of  handling  other  tutoring  content. 

In  this  paper,  we  present  the  effect  of  using  MTT  in  learning  PoA  by  university  students.  We 
found  our  participants  improved  in  accuracy  and,  simultaneously,  reduced  the  time  to 
determine  PoA.  Although  the  participants  may  not  have  had  a  need  to  master  marksmanship 
for  purposes  outside  of  this  study,  this  report  still  provides  insights  about  the  effectiveness  of 
using  MTT,  along  with  D2P,  and  points  to  directions  for  future  improvements. 

In  a  previous  study  (Ritter  et  al.,  2013;  Yeh  &  Ritter,  2012)  with  a  previous  version  of  the 
tutor,  we  examined  how  well  D2P/MTT  improved  tutees’  abilities.  In  the  previous  study 
tutees  improved  their  abilities  from  20%  to  47%  as  measured  by  time  of  retrieving  PoA.  The 
previous  study  found  an  effect  size  (a  way  of  measuring  tutoring  impact)  of  1 .48  compared  to 
baseline.  We  believe  that  the  Marine  Corps  currently  does  not  use  a  tutoring  approach  that 
leads  to  any  procedural  learning  with  respect  to  hitting  moving  targets,  although  the  tutoring 
could  lead  to  some  knowledge  about  the  declarative  elements. 

During  and  after  the  previous  study  (Yeh  &  Ritter,  2012)  we  examined  how  the  tutor  could  be 
improved,  and  created  a  revised  version  of  the  tutor.  We  made  approximately  100  corrections 
and  additions  to  the  tutor. 

In  this  paper,  we  present  the  effect  of  using  this  revised  MTT  in  learning  PoA  by  university 
students.  We  found  our  participants  improved  in  accuracy  and  simultaneously  reduced  the 
time  it  took  them  to  determine  PoA.  Although  the  participants  may  not  have  had  a  need  to 
master  marksmanship  for  purposes  outside  of  this  study,  this  report  still  provides  insights 
about  the  effectiveness  of  using  MTT,  along  with  D2P,  and  points  to  directions  for  future 
improvements. 
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2.0  Method 


2.1  Materials 

To  measure  learning,  we  used  a  pre-test  and  a  post-test  that  each  consisted  of  24  multiple 
choice  questions  without  feedback  asking  participants  to  judge  angle,  range,  speed,  or  points- 
of-aim  to  hit  a  moving  target,  using  videos  showing  targets  running  or  walking  (varied  target 
speed),  at  25,  50,  or  100  meters  (varied  range),  and  at  45  or  90  degree  angles  (varied  angle). 
They  each  had  equal  number  of  questions  of  each  type.  Figure  1  shows  a  screen  shot  of  an 
example  pre/post  test  question. 


Question  2/24 


What  is  Ae  speed  of  this  mo\ing  target?  Play  Video] 

A.  5  mph 

B.  10 mph 

I - 

[  Next  | 

Figure  1.  An  example  video  pre-  and  post-test  video  question. 

We  used  the  D2P/MTT  tutor,  version  of  1.4.1  of  27  November  2012.  It  contains  141  pages  of 
instructional  material — roughly  two  to  three  hours  of  instructional  materials,  including  text, 
still  images,  audio  material,  videos,  and  quizzes.  Figure  2  shows  an  example  screen  shot  of  a 
video  included  in  the  tutor,  and  Figure  3  shows  an  example  training  test  included  in  the  tutor. 
The  specific  topics  covered  are,  in  order,  pre-qualification  exercises  to  establish  a  baseline 
about  user  knowledge,  an  assessment  of  user  ability  to  estimate  a  target’s  range,  speed,  and 
angle,  a  points-of-aim  training  section,  an  observation  and  engagement  section,  a  qualification 
exam  to  assess  what  the  user  has  learned,  and  a  final  practice  module  to  reinforce  PoA 
estimation.  The  RUI  keystroke  and  mouse  logger  (Kukreja,  Stevenson,  &  Ritter,  2006; 
Morgan,  Cheng,  Pike,  &  Ritter,  2013)  was  running  on  the  laptop. 
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Challenges  Associated  with  Moving  Targets 

Because  the  target  will  most  likely  continue  moving  during  the  bullet's  flight,  hitting  a  moving  target  in  many  instances  requires  compensating  for  that 
movement. 


Figure  2.  Screen  shot  of  an  instructional  video  used  in  MTT. 


Figure  3.  A  screenshot  of  a  question  used  in  the  range  assessment  section  of  MTT. 


2.2  Subjects 

There  were  23  subjects  who  were  university  students  18  to  27  years  of  age.  They  were 
recruited  with  a  flyer  asking  for  subjects  to  test  software  for  the  Marine  Corps.  They  were 
paid  $7  per  hour  to  the  nearest  half  hour  rounded  up. 

2.3  Design 

The  study  design  was  a  within-subject  design  across  question  type.  Each  subject  was 
measured  by  their  performance  of  before  and  after  the  experiment.  The  assessment  consisted 
4  types  of  questions:  judging  speed,  angle,  distance,  and  retrieved  PoA. 

2.4  Procedure 

Participants  used  D2P/MTT  in  a  quiet  room  on  a  Windows  XP  laptop,  with  the  experimenter 
in  an  adjacent  room  separated  by  a  partially  closed  door.  Participants  were  encouraged  to  ask 
the  experimenter  if  they  needed  any  assistance. 


3 


16  Aug  2013 


Upon  arrival,  each  participant  was  first  told  that  they  would  be  working  through  a  tutor  that 
was  designed  to  teach  them  how  to  shoot  moving  targets,  and  that  this  tutor  was  intended  to 
be  used  by  U.S.  Marines  in  the  future.  They  were  told  that  they  would  take  a  short  break  after 
1.5  hours.  Then  a  consent  form  was  presented  to  them  that  outlined  the  risks  of  the 
experiment,  followed  by  a  two-page  questionnaire  about  basic  personal  information  as  well  as 
past  experience  shooting  moving  targets,  simulated  or  real. 

Participants  then  took  a  multiple  choice  24  question  pre-test  without  feedback  asking  them  to 
judge  angle,  range,  speed,  or  points-of-aim  to  hit  a  moving  target,  using  videos  showing 
targets  running  or  walking  (varied  target  speed),  at  25,  50,  or  100  meters  (varied  range),  and 
at  45  or  90  degree  angles  (varied  angle). 

Participants  then  began  using  the  D2P/MTT.  1.5  hours  after  the  participant  arrived,  they  were 
asked  to  conclude  the  section  they  were  working  on  to  take  a  short  break  (approximately  5  to 
10  minutes).  After  the  break,  they  resumed  using  the  tutor.  After  finishing  the  MTT,  they  took 
a  multiple  choice  post-test  without  feedback  that  consisted  of  pre-test  questions  reordered  to 
assess  MTT’s  effectiveness  in  teaching  them  how  to  retrieve  PoA  for  moving  targets. 

3.0  Results 

All  23  participants  completed  the  D2P/MTT  tutor  training.  To  compute  time  on  task  we 
measured  the  difference  between  the  starting  time  of  pre-test  and  the  starting  time  of  post-test 
(both  are  recorded  in  RUI)  and  report  it  as  the  time  on  task.  The  actual  time  of  using  MTT  is, 
therefore,  shorter  than  what  we  report  in  this  paper.  A  pre-test  takes  approximately  10  to  15 
minutes.  The  most  time  any  of  the  subjects  took  was  3  hours  24  minutes  in  total.  The  average 
time  with  the  tutor  was  1  hour  55  minutes,  and  the  SD  was  27  minutes. 

Among  all  participants,  seven  required  some  form  of  assistance  to  deal  with  frozen  pages  in 
the  tutor.  The  experimenter  resolved  each  issue  within  two  minutes,  typically  by  restarting  the 
tutor,  and  participants  were  able  to  continue  using  the  tutor  in  all  but  one  case.  In  that  one 
case,  S12  encountered  a  frozen  on  page  129  that  the  experimenter  was  unable  to  fix;  they  had 
already  completed  over  90  percent  of  the  tutor  and  the  only  section  they  did  not  complete  was 
a  practice  section  with  a  format  identical  to  previous  practice  sections.  This  subject  was  not 
excluded  from  analyses. 

The  average  score  of  all  questions  in  the  pre-test  (24  questions  total)  is  3.43  (SD  =  2.46).  In 
addition,  the  average  PoA  score  (out  of  10)  is  2.39  (SD  =  1.34).  After  completing  D2P/MTT, 
the  average  of  post-test  (24  questions  total)  score  is  22.70  (SD  =  1.87)  and  the  average  of  PoA 
score  is  8.91  (SD=1.38). 

Table  1  shows  that  our  participants  improved  in  all  categories  after  completing  the  tutor. 
Overall  and  for  the  PoA  questions  the  differences  are  reliable,  as  shown  in  the  last  column. 
The  greatest  improvement  is  in  the  PoA  category  that  most  people  do  not  have  prior 
knowledge  and  is  not  one  of  skills  we  use  daily.  Subjects  more  than  tripled  their  ability  to 
judge  PoA.  For  the  non-PoA  categories,  although  the  participants  scored  high  in  the  pre-test, 
their  post-test  scores  generally  showed  greater  consistency  (lower  SDs)  across  all  types  of 
questions. 
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Table  1.  Scores  on  the  pre-  and  post-test.  (N=23). 


Average  Score 

Category 

Pre-test 

Post-test 

T 

P 

(#  of  questions) 

(SD) 

(SD) 

Overall  (24) 

13.43  (2.46) 

22.70  (1.87) 

4.24 

.0003 

Distance  (6) 

4.13  (1.25) 

5.84  (0.63) 

1.73 

.0981 

Speed (4) 

3.52  (0.67) 

3.91  (0.29) 

0.76 

.4580 

Angle  (4) 

3.39  (0.89) 

4.00  (0.00) 

0.97 

.3429 

PoA  (10) 

2.39  (1.34) 

8.91  (1.38) 

4.79 

.0001 

Table  2  shows  on  average,  participants  performed  the  post- test  much  faster  than  the  pre-test. 
We  used  the  starting  time  (ti)  that  the  experimenter  pressed  Fll  to  maximize  the  browser 
window  and  handed  the  task  over  to  the  participants  and  the  time  (t2)  that  the  participants  to 
the  participants  returned  the  control  to  the  experimenter  who  pressed  Fll  again  to  calculate 
the  duration  (t2-ti).  Individually,  all  but  one  (from  333.6  in  pre-test  to  399.0  in  post-test) 
participants  shortened  the  duration  of  the  post-test,  compared  to  the  duration  of  the  pre-test. 


Table  2.  Average  time  to  complete  pre-test  and  post-tests  (N=23) 


Pre-test  (SD) 

Post-test  (SD) 

T 

P 

Average  time  (seconds) 

435.1  (124.17) 

240.5  (53.13) 

2.04 

.0538 

We  would  like  to  compute  the  effect  size  of  this  tutor.  To  do  this,  we  need  a  control  group 
showing  what  learning  occurs  if  you  do  not  use  the  tutor  or  what  learning  occurs  during  the 
current  training.  Current  Marine  Corps  policy  is  to  teach  how  to  shoot  moving  targets  based 
on  a  single,  annual,  1-hour  PowerPoint  lecture.  What  we  know  about  learning  procedural 
skills,  all  that  we  know  about  learning  procedural  skills,  predicts  that  this  lecture, 
theoretically,  does  not  lead  to  any  procedural  learning  (Anderson,  1995;  Anderson,  Greeno, 
Kline,  &  Neves,  1981;  Kim,  Ritter,  &  Koubek,  2013;  Larkin,  McDermott,  Simon,  &  Simon, 
1980;  Newell  &  Rosenbloom,  1981).  So,  to  compute  effect  size  we  use  the  initial  scores  as 
the  control  group.  This  approach  may  have  to  be  modified  when  using  Marines  because  we 
may  find  that  they  do  learn  from  a  lecture  without  any  practice,  quiz,  or  training,  but  it  is 
highly  unlikely.  We  may  have  to  do  this  condition  as  future  work,  because  some  informal 
reviewers  have  suggested  that  such  learning  is  perhaps  possible. 

So,  the  t-test  values  in  Table  1  provide  initial  effect  sizes  for  the  tutor’s  components.  Overall, 
for  PoA  and  for  all  the  questions,  the  tutor  has  an  effect  size  greater  than  4  over  all  questions 
and  for  the  most  important  type  of  question,  point  of  aim. 
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The  results  from  the  post-test  are  exceptionally  high,  which  suggests  a  ceiling  effect  occurs. 
Therefore,  including  the  standard  deviation  from  the  post-test  would  generate  a  larger  effect 
size  because  of  small  standard  deviation.  If  we  choose  to  use  the  more  conservative  standard 
deviation  (from  the  pre-test)  to  compute  and  report  our  effect  size  of  the  tutor  (Olejnik  & 
Algina,  2000),  we  still  have  an  effect  size  of  3.7  overall  and  4.8  for  the  PoA  questions.  Using 
either  calculation  the  effect  size  is  very  high,  and  we  will  take  up  its  import  in  the  next 
section. 

4.0  Discussion  and  Conclusions 

The  result  of  our  experiment  indicates  that  using  the  revised  MTT  is  better  than  the  previous 
version  (Yeh  &  Ritter,  2012),  and  that  the  revised  version  can  significantly  improve  the 
ability  to  judge  PoA  by  practicing  the  other  common,  component  skills,  i.e.  judging  distance, 
speed,  and  angle,  and  retrieving  PoA.  We  attribute  the  improvement  to  three  features  of  the 
MTT:  (1)  practicing  declarative  knowledge  explicitly,  (2)  doing  this  enough  to  increase  the 
chance  of  proceduralizing  these  skills,  and  (3)  providing  heuristics  for  learning  these  skills. 

The  design  goal  of  MTT  is  to  promote  procedural  declarative  knowledge  through  repetitive 
reinforcement.  The  MTT  fulfills  this  design  goal  and  helps  learners  improve  their  proficiency 
at  judging  PoA.  Learners  receive  plenty  of  opportunities  to  practice  the  skills  the  tutor  is 
intended  to  teach  them.  Each  practice  increases  the  chance  of  proceduralization.  However,  we 
do  not  know  whether  the  learners  in  fact  proceduralize  the  declarative  knowledge.  We  only 
know  their  performance  does  increase  after  the  tutor  training.  A  measurement  of  reaction  time 
is  required  for  this  type  of  analysis. 

Essentially  we  need  learners  to  estimate  PoA  quickly.  In  the  MTT,  we  teach  learners  to  use 
heuristics  such  as  the  height  of  runner’s  knee  to  estimate  their  speed,  and  in  turn  detennine  the 
PoA.  Our  results  also  indicate  that  the  instructional  strategies  of  MTT  are  effective  with  an 
effect  size  greater  than  3.7.  This  effect  is  almost  twice  as  large  as  the  largest  reported  (1.95) 
in  a  recent  review  of  intelligent  tutoring  systems  (VanLehn,  2011),  and  larger  than  the 
previous  version  of  the  tutor  (Ritter  et  ah,  2013).  This  is  partly  due  to  the  baseline  we  chose, 
which  is  no  learning,  which  is  the  result  of  this  otherwise  essential  skill  currently  not  being 
trained  by  the  Marine  Corps.  The  large  effect  size  is,  of  course,  also  partly  due  to  the  tutor 
tutoring  and  getting  many  learners  to  the  ceiling  in  the  tutor. 

This  study  is  our  attempt  to  understand  whether  MTT  is  useful  and  usable.  Throughout  the 
process  of  design,  deployment,  and  evaluation,  issues  related  to  the  system  and  to  its  usability 
were  fixed  after  each  pilot  test  and  we  are  still  improving  it.  The  major  contribution  of  this 
study  is  to  demonstrate  that,  with  our  tutoring  architecture,  D2P,  creating  useful  tutors  can  be 
efficient  and  require  relatively  minimal  system  training. 

We  are  planning  additional  tests  of  the  MTT.  First  we  want  to  compare  the  MTT  with  a 
human  instructor  or  the  existing  field  manual.  This  may  have  to  be  tested  by  using  just  the 
current  instructional  material  used  by  the  Marine  Corps.  Second  we  want  to  measure  memory 
retention  in  both  the  MTT  and  human  instructor  situation.  If  we  have  enough  resources  and 
support,  testing  to  see  whether  learners  can  transfer  their  PoA  knowledge  to  actual  firing 
situations  would  allow  us  to  quantify  the  real-world  usefulness  of  the  MTT. 

We  will  also  test  whether  instructional  designers  or  people  who  do  not  possess  deep 
understanding  of  D2P  architecture  can  create  an  effective  tutor  or  not.  Our  intention  is  to 
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build  a  set  of  toolkits  for  people  who  are  outside  of  the  cognitive  science/modeling  area  of 
study. 

In  terms  of  the  tutoring  work,  we  will  continue  to  use  and  improve  our  tools,  including  MTT 
and  D2P  in  other  appropriate  learning  settings. 


References 

Anderson,  J.  R.  (1995).  Learning  and  memory.  New  York,  NY:  John  Wiley  and  Sons. 

Anderson,  J.  R.,  Greeno,  J.  G.,  Kline,  P.  J.,  &  Neves,  D.  M.  (1981).  Acquisition  of  problem-solving 
skill.  In  J.  R.  Anderson  (Ed.),  Cognitive  skills  and  their  acquisition.  Hillsdale,  NJ:  Erlbaum. 

Kim,  J.  W.,  Ritter,  F.  E.,  &  Koubek,  R.  J.  (2013).  An  integrated  theory  for  improved  skill  acquisition 
and  retention  in  the  three  stages  of  learning.  Theoretical  Issues  in  Ergonomics  Science,  14(  1),  22- 
37. 

Kukreja,  U.,  Stevenson,  W.  E.,  &  Ritter,  F.  E.  (2006).  RUI — Recording  User  Input  from  interfaces 
under  Windows  and  Mac  OS  X.  Behavior  Research  Methods,  38(4),  656-659. 

Larkin,  J.  H.,  McDermott,  J.,  Simon,  D.  P.,  &  Simon,  H.  A.  (1980).  Expert  and  novice  performance  in 
solving  physics  problems.  Science,  208,  1335-1342. 

Morgan,  J.  H.,  Cheng,  C.-Y.,  Pike,  C.,  &  Ritter,  F.  E.  (2013).  A  design,  tests,  and  considerations  for 
improving  keystroke  and  mouse  loggers.  Interacting  with  Computers,  25(3),  242-258. 

Newell,  A.,  &  Rosenbloom,  P.  S.  (1981).  Mechanisms  of  skill  acquisition  and  the  law  of  practice.  In  J. 
R.  Anderson  (Ed.),  Cognitive  skills  and  their  acquisition  (pp.  1-51).  Hillsdale,  NJ:  Erlbaum. 

Olejnik,  S.,  &  Algina,  J.  (2000).  Measures  of  effect  size  for  comparative  studies:  Applications, 
interpretations,  and  limitations.  Contemporary  Educational  Psychology,  25(3),  241-286. 

Ritter,  F.  E„  Yeh,  K.-C.,  Cohen,  M.  A.,  Weyhrauch,  P.,  Kim,  J.  W„  &  Hobbs,  J.  N.  (2013). 

Declarative  to  Procedural  Tutors:  A  family  of  cognitive  architecture-based  tutors.  In  Proceedings 
of  the  22nd  Conference  on  Behavior  Representation  in  Modeling  and  Simulation,  BRIMS2013- 
2127.  BRIMS  Society:  Centerville,  OH. 

VanLehn,  K.  (201 1).  The  relative  effectiveness  of  human  tutoring,  intelligent  tutoring  systems,  and 
other  tutoring  systems.  Educational  Psychologist,  46(4),  197-221. 

Yeh,  K.-C.,  &  Ritter,  F.  E.  (2012).  An  initial  evaluation  of  the  D2P/MTT,  a  computer-based, 

Declarative  to  Procedural  (D2P)  theory  driven  moving  target  tutor  (Tech.  Report  No.  ACS  2012- 
1).  Applied  Cognitive  Science  Lab,  College  of  Information  Sciences  and  Technology,  Penn  State. 


7 


16  Aug  2013 


